IN THE CLAIMS : 

Please cancel Claims 58, 59, 73, and 74 without prejudice to or disclaimer of 
the subject matter recited therein. 

Please amend Claims 35, 71 and 72 and add Claims 75-77 as follows. 

1 . (Withdrawn) An image processing method for describing a real image as a 
virtual object using space data formed based on a photo image of the real object, and 
generating a virtual image of the virtual object using the space data, comprising the steps 
of: 

capturing the photo image of the real object from a real camera viewpoint 
position while illuminating the real object by an illumination light source placed at a real 
illumination position; 

converting the phot image into space data in the form of an object; and 
storing the space data in a memory together with various illumination 
conditions at the real illumination position to allow a search at a later time. 

2. (Withdrawn) The method according to claim 1, wherein the illumination 
position of the illumination light source is set at a plurality of different positions, and 
position information and an illumination condition at each illumination position are stored 
in the memory together with corresponding space data. 
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3. (Withdrawn) The method according to claim 1, wherein the camera 
viewpoint position is set at a plurality of different positions, and space data obtained at 
each viewpoint position is stored together with the viewpoint position information. 



4. (Withdrawn) The method according to claim 1, wherein the space data is 
ray space theory data. 

5. (Withdrawn) The method according to claim 1, wherein the illumination 
condition is varied in a plurality of ways per illumination position, and space data 
corresponding to the individual illumination conditions are stored in the memory. 

6. (Withdrawn) An image processing method for describing a real image a s 
virtual object using space data formed based on a photo image of the real object, and 
generating a virtual image of the virtual object using the space data, comprising the steps 
of: 

storing space data generated from a photo image together with a real camera 
viewpoint position, and an illumination position and illumination condition of a real 
illumination light source; 

generating coordinates of a virtual image of an object at a user viewpoint 
position on the basis of the real camera viewpoint position and the space data; and 

correcting pixel values of the virtual image on the basis of the real illumination 
position and real illumination condition of the real illumination light source, and a virtual 
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illumination position and virtual illumination condition set for a virtual illumination light 
source. 

7. (Withdrawn) The method according to claim 6, wherein the illumination 
position of the real illumination light source is set at a plurality of different positions, and 
position information and an illumination condition at each illumination position are stored 
in the memory together with corresponding space data. 

8. (Withdrawn) The method according to claim 6, wherein the real camera 
viewpoint position is set at a plurality of different positions, and space data obtained at 
each viewpoint position is stored together with the viewpoint position information. 

9. (Withdrawn) The method according to claim 6, wherein the space data is 
ray space theory data. 

10. (Withdrawn) The method according to claim 6, further comprising the step 
of changing and setting an illumination condition of the virtual illumination light source to 
be an arbitrary value, and wherein the correction step includes the step of correcting the 
pixel values in accordance with the changed illumination condition. 

11. (Withdrawn) The method according to claim 6, wherein when a plurality 
of virtual light sources are ON at the same time, a plurality of correction results which are 
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corrected in accordance with the respective virtual illumination light sources are added for 
one pixel position to obtain a final pixel value. 

12. (Withdrawn) The method according to claim 6, wherein when some of a 
plurality of virtual light sources are ON at the same time, a plurality of correction results 
which are corrected in accordance with the respective ON virtual illumination light sources 
are added for one pixel position to obtain a final pixel value. 

13. (Withdrawn) The method according to claim 6, further comprising the 

steps of : 

determining if the corrected pixel value saturates; and 

virtually changing illumination conditions of some or all of the virtual light 

sources in a direction to decrease luminance or illuminance of an illumination when the 

pixel value saturates, and 

wherein the correction step is executed again in accordance with the changed 

illumination condition. 

14. (Withdrawn) The method according to claim 6, wherein the illumination 
condition includes information indicating whether or not an illumination is turned on. 

15. (Withdrawn) The method according to claim 6, wherein the illumination 
condition includes illuminance or luminance. 
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16. (Withdrawn) The method according to claim 6, wherein the illumination 
condition includes a relative position between the virtual light source and object. 

17. (Withdrawn) The method according to claim 6, wherein the illumination 
condition includes color of illumination. 

18. (Withdrawn) An image processing apparatus for describing a real image as 
a virtual object using space data formed based on a photo image of the real object, and 
generating a virtual image of the virtual object using the space data, comprising: 

means for capturing the photo image of the real object from a real camera 
viewpoint position while illuminating the real object by an illumination light source placed 
at a real illumination position; 

conversion means for converting the photo image into space data in the form of 
an object for each pixel; and 

means for storing the space data in a memory together with various 
illumination conditions at the real illumination position to allow a search at a later time. 

19. (Withdrawn) The apparatus according to claim 18, wherein the 
illumination position of the illumination light source is set at a plurality of different 
positions, and position information and an illumination condition at each illumination 
position are stored in the memory together with corresponding space data. 
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20. (Withdrawn) The apparatus according to claim 18, wherein the camera 
viewpoint position is set at a plurality of different positions, and space data obtained at 
each viewpoint position is stored together with the viewpoint position information. 



21 . (Withdrawn) The apparatus according to claim 18, wherein the space data 
is ray space theory data. 

22. (Withdrawn) The apparatus according to claim 18, wherein the 
illumination condition is varied in a plurality of ways per illumination position, and space 
data corresponding to the individual illumination conditions are stored in the memory. 

23. (Withdrawn) An image processing apparatus for describing a real image as 
a virtual object using space data formed based on a photo image of the real object, and 
generating a virtual image of the virtual object using the space data, comprising: 

means for storing space data in the form of an object for each pixel of a photo 
image together with a real camera viewpoint position, and an illumination position and 
illumination condition of a real illumination light source; 

generation means for generating coordinates of a virtual image of an object at a 
user viewpoint position on the basis of the real camera viewpoint position and the space 
data; and 

correction means for correcting pixel values of the virtual image on the basis of 
the real illumination position and real illumination condition of the real illumination light 
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source, and a virtual illumination position and virtual illumination condition set for a 
virtual illumination light source. 

24. (Withdrawn) The apparatus according to claim 23 , wherein the 
illumination position of the real illumination light source is set at a plurality of different 
positions, and a position information and an illumination condition at each illumination 
position are stored in the memory together with corresponding space data. 

25. (Withdrawn) The apparatus according to claim 23, wherein the real camera 
viewpoint position is set at a plurality of different positions, and space data obtained at 
each viewpoint position is stored together with the viewpoint position information. 

26. (Withdrawn) The apparatus according to claim 23, wherein the space data 
is ray space theory data. 

27. (Withdrawn) The apparatus according to claim 23, further comprising 
means for changing and setting an illumination condition of the virtual illumination light 
source to be arbitrary values, and wherein said correction means corrects the pixel values in 
accordance with the changed illumination condition. 

28. (Withdrawn) The apparatus according to claim 23, wherein when a 
plurality of virtual light sources are ON at the same time, a plurality of correction results 
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which are corrected in accordance with the respective virtual illumination light sources are 
added for one pixel position to obtain a final pixel value. 

29. (Withdrawn) The apparatus according to claim 23, wherein when some of 
a plurality of virtual light sources are ON at the same time, a plurality of correction results 
which are corrected in accordance with the respective ON virtual illumination light sources 
are added for one pixel position to obtain a final pixel value. 

30. (Withdrawn) The apparatus according to claim 23, further comprising: 
means for determining if the corrected pixel value saturates; and 

means for virtually changing illumination conditions of some or all of the 
virtual light sources in a direction to decrease luminance or illuminance of an illumination 
when the pixel value saturates, and 

wherein said correction means is executed again in accordance with the 
changed illumination condition. 

3 1 . (Withdrawn) The apparatus according to claim 23, wherein the 
illumination condition includes information indicating whether or not an illumination is 
turned on. 

32. (Withdrawn) The apparatus according to claim 23, wherein the 
illumination condition includes illuminance or luminance. 
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33. (Withdrawn) The apparatus according to claim 23, wherein the 
illumination condition includes a relative position between the virtual light source and 
object. 



34. (Withdrawn) The apparatus according to claim 23, wherein the 
illumination condition includes color of illumination. 



35. (Currently Amended) An image processing method for generating a 
shadow image of a virtual object itself in a virtual space, comprising the steps of: 

sensing an image of a real object by a camera while illuminating the real object 
corresponding to the virtual object from by a light source at a predetermined illumination 
position; 

extracting a silhouette of the real object from the image of the real object; 
adding preferred image information as a shadow to pixels in the silhouette; and 
storing the silhouette image together with position information of the light 
source, the silhouette image being able to be found by search and retrieved at a later time. 



36. (Original) the method according to claim 35, wherein the preferred image 
information includes a predetermined black pixel value and a predetermined transparency 
value. 



37. (Original) The method according to claim 35, wherein the silhouette image 
undergoes a blur process. 
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38. (Original) The method according to claim 37, wherein the degree of blur is 
changed in correspondence with distance from the virtual object to a virtual light source. 

39. (Original) The method according to claim 35, wherein a viewpoint 
position of the camera matches the position of the illumination light source. 

40. (Original) An image processing method for generating a shadow image of 
a virtual object itself in a virtual space, comprising the steps of: 

storing space data of the virtual object, a shadow image of a real object 
corresponding to the virtual object, and a position of an illumination light source upon 
forming the shadow image in a predetermined memory; 

reading out the shadow image from the memory in accordance with a position 
of a virtual illumination, and a relative position of the virtual object; and 

mapping the readout shadow image on a predetermined mapping plane. 

41 . (Original) The method according to claim 40, wherein the mapping plane 
is determined on the basis of a bounding box of the virtual object. 

42. (Original) The method according to claim 40, further comprising the step 
of changing and setting an illumination condition of the virtual illumination light source to 
be an arbitrary value, and wherein the correction step includes the step of correcting the 
pixel values in accordance with the changed illumination condition. 
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43. (Original) The method according to claim 40, wherein when a plurality of 
virtual light sources are ON at the same time, a plurality of correction results which are 
corrected in accordance with the respective virtual illumination light sources are added for 
one pixel position to obtain a final pixel value. 

44. (Original) The method according to claim 40, wherein when some of a 
plurality of virtual light sources are ON at the same time, a plurality of correction results 
which are corrected in accordance with the respective ON virtual illumination light sources 
are added for one pixel position to obtain a final pixel value. 

45. (Original) The method according to claim 40, wherein when the 
illumination is located at a high-level position, a mapping plane with a simple shape is 
generated. 

46. (Original) An image processing apparatus for generating a shadow image 
of a virtual object itself in a virtual space, comprising: 

means for sensing an image of a real object by a camera while illuminating the 
real object corresponding to the virtual object from a light source at a predetermined 
illumination position; 

means for extracting a silhouette of the real object from the image of the real 

object; 

means for adding preferred image information as a shadow to pixels in the 
silhouette; and 
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means for storing the edge image together with position information of the 
light source, the silhouette image being able to be found by search and retrieved at a later 
time. 

47. (Original) The apparatus according to claim 46, wherein the preferred 
image information includes a predetermined black pixel value and a predetermined 
transparency value. 

48. (Original) The apparatus according to claim 46, wherein when the 
viewpoint position of the camera is different from the position of the light source, the 
silhouette image is corrected by re-projecting the silhouette image to have the position of 
the light source as a viewpoint position. 

49. (Original) The apparatus according to claim 46, further comprising means 
for performing a blur process of the silhouette image. 

50. (Original) The apparatus according to claim 49, wherein the degree of blur 
is changed in correspondence with distance from the virtual object to a virtual light source. 

51. (Original) The apparatus according to claim 46, wherein a viewpoint 
position of the camera matches the position of the illumination light source. 



- 13- 



52. (Original) An image processing apparatus for generating a shadow image 
of a virtual object itself in a virtual space, comprising the steps of: 

means for storing space data of the virtual object, a shadow image of a real 
object corresponding to the virtual object, and a position of an illumination light source 
upon forming the shadow image in a predetermined memory; 

means for reading out the shadow image from the memory in accordance with 
a position of a virtual illumination, and a relative position of the virtual object; and 

means for mapping the readout shadow image on a predetermined mapping 

plane. 

53. (Original) the apparatus according to claim 52, wherein the mapping plane 
is determined on the basis of a bounding box of the virtual object. 

54. (Original) The apparatus according to claim 52, further comprising means 
for changing and setting an illumination condition of the virtual illumination light source to 
be an arbitrary value, and wherein said correction means corrects the pixel values in 
accordance with the changed illumination condition. 

55. (Original) The apparatus according to claim 52, wherein when a plurality 
of virtual light sources are ON at the same time, a plurality of correction results which are 
corrected in accordance with the respective virtual illumination light sources are added for 
one pixel position to obtain a final pixel value. 
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56. (Original) The apparatus according to claim 52, wherein when some of a 
plurality of virtual light sources are ON at the same time, a plurality of correction results 
which are corrected in accordance with the respective ON virtual illumination light sources 
are added for one pixel position to obtain a final pixel value. 

57. (Original) The apparatus according to claim 52, wherein when the 
illumination is located at a high-level position, a mapping plane with a simple shape is 
generated. 

58. (Cancelled) 

59. (Cancelled) 

60. (Withdrawn) An image processing apparatus for mixing a virtual space 
with a real space illuminated by a real illumination device, and presenting the mixed space, 
comprising: 

change means for changing an illumination condition of the illumination 

device; 

first setting means for setting an illumination program parameter value 
corresponding to a value of the changed illumination condition; 

rendering means for rendering a virtual image using a predetermined 
illumination condition program parameter, and mixing the virtual image with the real 
space; and 
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second setting means for setting, in a rendering routine of the virtual image of 
said rendering means, the illumination condition program parameter set by said setting 
means in correspondence with the illumination condition changed by said change means. 



61 . (Withdrawn) The apparatus according to claim 60, wherein the 
illumination condition includes one of illuminance, color, and an illumination direction. 

62. (Withdrawn) The apparatus according to claim 60, wherein said change 
means comprises GUI means operated by a user to display control buttons for graphically 
changing illuminance, color, and an illumination direction of the illumination device on a 
display screen. 

63. (Withdrawn) The apparatus according to claim 60, wherein said change 
means comprises GUI means operated by a user, and said GUI means displays an image 
that simplifies the illumination device and allows the user to instruct to change a given 
illumination condition by designating a portion to be adjusted in the image using a pointing 
device. 

64. (Withdrawn) The apparatus according to claim 60, wherein said first 
setting means includes means for reading out a parameter pre-stored in a predetermined 
memory. 
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65. (Withdrawn) An image processing method for mixing a virtual space with 
a real space illuminated by a real illumination device, comprising: 

the step of changing an illumination condition of the illumination device; 

the first setting step of setting an illumination program parameter value 
corresponding to a value of the changed illumination condition; 

the step of rendering a virtual image using a predetermined illumination 
condition program parameter, and mixing the virtual image with the real space; and 

the second setting step of setting, in a rendering routine of the virtual image, 
the illumination condition program parameter set in correspondence with the illumination 
condition changed in the change step. 

66. (Withdrawn) The method according to claim 65, wherein the first setting 
step includes the step of reading out a parameter pre-stored in a predetermined memory. 

67. (Withdrawn) The method according to claim 65, wherein the change step 
includes the step of displaying a GUI operated by a user to display control buttons for 
graphically changing illuminance, color, and an illumination direction of the illumination 
device on a display screen. 

68. (Withdrawn) The method according to claim 65, wherein the change step 
includes the step of displaying a GUI operated by a user, and the GUI displays an image 
that simplifies the illumination device and allows the user to instruct to change a given 
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illumination condition by designating a portion to be adjusted in the image using a pointing 
device. 

69. (Withdrawn) A storage medium that stores a control program for making a 
computer execute an image process for describing a real image as a virtual object using 
space data formed based on a photo image of the real object, and generating a virtual image 
of the virtual object using the space data, said control program including: 

a code of the capture step of capturing the photo image of the real object from a 
real camera viewpoint position while illuminating the real object by an illumination light 
source placed at a real illumination position; 

a code of the conversion step of converting the photo image into space data in 
the form of an object; and 

a code of the storage step of storing the space data in a memory to allow a 
search together with various illumination conditions at the real illumination position. 

70. (Withdrawn) A storage medium that stores a control program for making a 
computer execute an image process for describing a real image as a virtual object using 
space data formed based on a photo image of the real object, and generating a virtual image 
of the virtual object using the space data, said control program including: 

a code of the storage step of storing space data generated from a photo image 
together with a real camera viewpoint position, and an illumination position and 
illumination condition of a real illumination light source; 
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a code of the generation step of generating coordinates of a virtual image of an 
object at a user viewpoint position on the basis of the real camera viewpoint position and 
the space data; and 

a code of the correction step of correcting pixel values of the virtual image on 
the basis of the real illumination position and real illumination condition of the real 
illumination light source, and a virtual illumination position and virtual illumination 
condition set for a virtual illumination light source. 

71. (Currently Amended) A storage medium that stores a control program for 
making a computer execute an image process for generating a shadow image of a virtual 
object itself in a virtual space, said control program including image process comprising 
the steps of : 

a code of the image sensing step of sensing acquiring an image of a real object 
sensed by a camera while illuminating the real object corresponding to the virtual object 
from a light source at a predetermined illumination position; 

a cod e of the ext r action step of extracting a silhouette of the real object form 
the image of the real object; 

a code of the addition step of adding preferred image information as a shadow 
to pixels in the silhouette; and 

a code of the storage step of storing the silhouette image together with position 
information of the light source, the silhouette image being able to be found by search and 
retrieved at a later time. 
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72. (Currently Amended) A storage medium that stores a control program for 
making a computer execute an image process for generating a shadow image of a virtual 
object itself in a virtual space, said control program including image process comprising 
the steps of : 

a cod e of th e sto r age step of storing space data of the virtual object, a shadow 
image of a real object corresponding to the virtual object, and a position of an illumination 
light source upon forming the shadow image in a predetermined memory; 

a code of th e r e ad-out step of reading out the shadow image form the memory 
in accordance with a position of a virtual illumination, and a relative position of the virtual 
object; and 

a code of the mapping step of mapping the readout shadow image on a 
predetermined mapping plane. 

73. (Cancelled) 

74. (Cancelled) 

75. (New) An image processing method for generating a shadow image of a 
virtual object itself in a virtual space, comprising the steps of: 

acquiring ray space data of the virtual object; 
setting a position of a virtual illumination; 

obtaining a shadow image of the virtual object, corresponding to the position of 
the virtual illumination; 
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calculating a shadow mapping plane based on a shape of a bounding box for 

the virtual object and the position of the virtual illumination; and 

mapping the shadow image on the shadow mapping plane, 

wherein the shadow image is generated by extracting a silhouette of a real 

object corresponding to the virtual object from an image obtained by sensing the real object 

from the position of the virtual illumination. 

76. (New) An image processing apparatus for generating a shadow image of a 
virtual object itself in a virtual space, said apparatus comprising: 

an acquiring unit configured to acquire ray space data of the virtual object; 

a setting unit configured to set a position of a virtual illumination; 

an obtaining unit configured to obtain a shadow image of the virtual object, 
corresponding to the position of the virtual illumination; 

a calculating unit configured to calculate a shadow mapping plane based on a 
shape of a bounding box for the virtual object and the position of the virtual illumination; 
and 

a mapping unit configured to map the shadow image on the shadow mapping 

plane; 

wherein the shadow image is generated by extracting a silhouette of a real 
object corresponding to the virtual object from an image obtained by sensing the real object 
from the position of the virtual illumination. 
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77. (New) A storage medium that stores a control program for making a 
computer execute an image process for generating a shadow image of a virtual object itself 
in a virtual space, said image process comprising the steps of: 

acquiring ray space data of the virtual object; 

setting a position of a virtual illumination; , j 

obtaining a shadow image of the virtual object, corresponding to the position of 

the virtual illumination; 

calculating a shadow mapping plane based on a shape of a bounding box for 

the virtual object and the position of the virtual illumination; and 

mapping the shadow image on the shadow mapping plane, 

wherein the shadow image is generated by extracting a silhouette of a real 

object corresponding to the virtual object from an image obtained by sensing the real object 

form the position of the virtual illumination. 
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